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Task 6 


Table 4,5 


Table 
11,12 


2.1.1 


Power use 
reductions 
for 
on/Standby 
mode for 
dual power 
supplies. 


While the consultant briefly notes the 
operation difficulties posed by the 
on/standby mode for redundant 
power supplies in data centers which 
use dual power feeds (separate, 
distinct power feeds to ensure 
availability), those difficulties and 
their impact on the stability of a data 
center configured with that type of 
power system are largely ignored in 
the discussions of potential savings. 
One manufacturer has removed the 
on/standby options on its systems 
because of strong feedback from 
multiple customers regarding the 
operational instability introduced to 
their data centers by the on/standby 
function. 


Provide a clear statement that redundant power supplies 
configured with an on/standby capability are not 
appropriate for some data center configurations and 
should not be a mandated solution for power supplies. 








Task 6 





1.1.3 





18 





ASHRAE A1 





Current server products are by and 
large capable of achieving ASHRAE 
Class 2 levels. Some storage 
products are A2 capable, while some 
are only A1 capable. See the 
ASHRAE website for specific 
temperature and humidity 
specifications. We have several 





Re-evaluate the energy savings available from setting 
ASHRAE standard requirements for server and storage 
products. The current savings are significantly 
overstated. Given the current state of the market (in 
2014/15) and available server and storage products, we 
believe that there are no or minimal potential energy 
savings available by moving to ASHRAE A1 or A2 
capable products, server products are largely designed 
for A2 environments and storage products for A1 or A2 
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concerns with the analysis. 


a. 


The consultant needs to get data 
on the current percentage of 
server and storage products 
currently on the market that are 
ASHRAE A1 and A2 capable. 
Savings calculations should only 
be based on the percentage of 
products that will increase their 
potential operating temperatures, 
not on the full count of unit sales. 


The EU Data Center Code of 
Conduct is currently specifying 
ASHRAE A2 as the level at which 
data centers should be operating. 


Data center operators purchasing 
equipment today are by and large 
purchasing equipment that is A2 
capable, with some storage 
equipment only capable of 
meeting A1 standards. Data 
center operators by and large 
have the ability to operate their 
data centers at A2 if they so 
choose. A2 should be the 
baseline comparison for energy 
savings calculations. If data 
center operators are not 
currently operating with the A2 
range, that decisions is not a 
result of the ASHRAE capabilities 





environments. Data center operating temperatures are 
managed by the data center operator — current 
products give them the ability to operate at ASHRAE 
A1 or A2 inlet temperature and humidity levels. 


Adjust the achievable PUE to a range of 1.4 to 1.8 in all 
but new, purpose built cloud data centers with largely 
homogeneous server, storage and network 
infrastructure — i.e. there are only 2 to 4 configurations 
of each of the 3 product types. The choice of a 
potential PUE of 1.25 grossly overstates the energy 
consumption reductions which can be achieved. The 
SNIA reference was taken out of context. The best 
available assessment of current data center PUEs in 
2014, the average of which is estimated at 1.70, is 
provided by Uptime at this url: 
http://www.datacenterknowledge.com/archives/2014/06 
/02/survey-industry-average-data-center-pue-stays- 
nearly-flat-four-years/ 











There is some uncertainty as to the actual current 
average PUE because the sample taken by Uptime may 
not be representative of the total installed base. There 
are also significant differences in PUE based on the 
geographic location of the datacenter. This link 
discusses that uncertainty. 
http:/Awww.computerworld.com/article/2496597/data- 
center/new-data-center-survey-shows-mediocre-results-for- 
energy-efficiency.html 


The consultant needs to get data on the percentage of 

current machine types that are designed for operations 
in ASHRAE A1 and A2 environments. Making a blanket 
statement that current products need to be brought up 
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of the equipment they are 
purchasing, it is a function of the 
ages of the individual equipment 
in the overall equipment 
inventory in their data center, the 
capability of their facilities 
systems and their ability to 
manage the risks of over- 


temperature in their data centers. 


Suggesting that you can drop 
from a PUE of 2 or greater to 1.2 
is unrealistic. Current 
experience suggests that a 
cooling optimized legacy data 
centers can achieve a PUE 
between 1.4 and 1.8 depending 
on the age and configuration of 
the data center, the age 
distribution of the IT equipment, 
and the installed (as opposed to 
potential) free cooling capability. 


It also needs to be noted that 
major temperature changes can 
be achieved only if new 
equipment is segregated into 
defined zones to take advantage 
of the higher allowable server 
inlet temperatures or where a 
data center is totally populated 
with A2 capable server and 
storage products. 





to the A1 standards is incorrect. For servers, DE 
believes that setting a requirement for A2 will take out 
the bottom of the market. For storage it is less clear. 
DE is working with its members to provide a 
compilation of what ASHRAE level current server and 
storage products are warranted to. 


4. The report needs to acknowledge that there are still 
potential reliability issues if storage equipment and 
storage media that are run in the high end of the 
ASHRAE A2 limits. Some currently available storage 
media is designed to operate reliably only in the 
ASHRAE A1 zone. 


These adjustments need to be reflected through sections 
1.2, 2.1, 2.2, and 2.3 and in all calculations and 
conclusions. 
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f. Some current storage products 
are designed to operate at A1 
levels and others at A2 levels. 
The consultant needs to 
understand what the distribution 
of products are in the market and 
determine if setting A2 as a 
minimum product standard is 
appropriate to remove the bottom 
of the market. However, savings 
estimates should be based on 
the current percentage products 
at A1, not the total universe of 
products sold. 








Task 6 





1.1.3 





14 





Increase in 
fan speed at 
higher server 
inlet 
temperatures 





For servers built and sold in 2012, it 
is correct that the energy 
consumption resulting for the 
increase in fan speed required at 
higher server inlet temperatures will 
largely negate the energy savings 
from reduced cooling requirements. 
It should be pointed out that the 
servers used in the base cases may 
not be designed for operation under 
ASHRAE A2 temperature conditions. 
The consultant needs to discuss the 
fact that equipment designed for A2 
operation should be designed that 
there are overall energy savings from 
running at the higher server inlet 
temperature. 





Modify the discussion to clearly state that IT products 
designed to operate under A2 conditions should provide 
net energy savings, though perhaps minimal, at the upper 
end of the ASHRAE A2 allowable temperature range. 
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The consultant should use 3 or 4 data center operation 
Task 6 Table 8 14 The selection of free cooling time scenarios: no free cooling and chillers/DX units > 10 years 
and operating PUE is completely old, new chiller with 40% average free cooling, and then an 
arbitrary and is representative of A2 system with 90% free cooling. This could correspond 
only 10-15% of the installed data to a PUE of 1.75; 1.45; and 1.1 respectively. The savings 
center base. The use of 1.25 as the calculations should then be calculated against a PUE of 
target PUE grossly overstates the 2.0 and the range of possible improvement should be 
potential energy savings realized by discussed. Current average PUE based on a survey by the 
driving improvements in the Uptime Institute for 2014 is 1.70. 
ASHRAE capabilities for server and http://www.datacenterknowledge.com/archives/2014/06/02/ 
storage products. survey-industry-average-data-center-pue-stays-nearly-flat- 
four-years/ 
These adjustments need to be reflected through sections 
1.2, 2.1, 2.2, and 2.3 and in all calculations and 
conclusions. 
Task 6 1.1.4 16 Power Digital Europe provided the Adjust the power use reduction range for the power 
management | consultant data that showed that x86 | preferred setting to 2-5% and recalculate the potential 














power management on a server 
released in 2013 showed only a 2-5% 
reduction in energy consumption 
between the power favoured and 
performance favoured power 
management settings. A 15-20% 
improvement was noted on resilient 
servers. Digital Europe can provide 
additional equipment data measured 
by the SERT test on x86 equipment. 
Calculation of possible energy 
consumption savings with a 10% - 





savings. These adjustments need to be reflected through 
sections 1.2, 2.1, 2.2, and 2.3 and in all calculations and 
conclusions. 
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relates to 
20% power use reduction using the 
power preferred mode will overstate 
the possible savings by a factor of 2 
to 5 times. 
Task 6 1.1.4 16 Power The discussion only mentions CPU Include memory and energy efficient Ethernet capabilities 
Management | power management. Memory power in the energy power management savings in the 
management and energy efficient discussion. 
Ethernet cards should also be 
considered and discussed. 

Task 6 1.2 15 Sectioning It is unclear why the blade analysis is | The Base Case analysis should be started as section 2. 
done in section 1.2 and the rack Section 2.1 should be the rack analysis, Section 2.2 blade 
analysis is done in 2.1. and section 2.3 storage. 

Task 6 Figure 22 | 24 Operational As discussed in comments The ASHRAE A1 and power management savings 

savings concerning sections 1.1.3 and 1.1.4, calculations need to be done and restated, including an 
the calculations shown in figure 21 analysis of the potential range of savings given different 
significantly overstate the average PUE and power management percent savings outcomes. 
savings and result in misleading As discussed above, the consultant needs to get current 
conclusions. Adjustments need to data on the percentage of products that are A1 and A2 
be made, as discussed, in the power | capable to properly assess potential savings. It is Digital 
management savings and the Europe’s belief that current products are either A1 or A2 
savings generated by making capable. 
products ASHRAE A1 or A2 capable 
(PUE savings) and different 
conditions need to be considered. 

Task 6 2.3.3 34 ASHRAE A2 | Digital Europe provided data Address the issue of increased drive failure in the analysis 
Operation for | indicating that storage products in this section. 
Storage operating at higher temperature 














ranges as will be be found at the 
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products higher temps for ASHRAE A1 and A2, 
are likely to experience increased 
failure rates. There is an economic 
cost that needs to be considered: 
increased maintenance costs and 
lost productivity. 
Task 6 1.3.1, 3.3 19-20, Storage Titanium power supplies for storage | Storage power supplies need to be analysed independent 
40 Product systems are largely not available, as | of server power supplies. 
Improvement | titanium power supplies have not 
Assessment | been qualified in storage products. Use gold supply as best available for OL-3 and above 














Titanium is achievable for server 
systems because they are single 
voltage power supplies. 


Titanium certified power supplies 
have been used in a very limited 
number of server products, but the 
qualification requirements for 
storage and server products can be 
different especially where the 
storage product uses a multi-output 
power supply 


Storage products often use a multi- 
voltage power supply. Storage is 
running 2 to 3 years behind server 
systems with regards to use of a 
given 80 plus power supply (or 
equivalent) efficiency level. 


Storage products typically require a 
larger output energy because of the 





storage products. 
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fact that spinning drives have a 
consistent power use with little 
variation between maximum and idle 
power use. 


The on/standby mode for redundant 
power supplies is difficult to apply to 
storage products. Per the 4 base 
cases Digital Europe provided to the 
consultant for storage products, the 
difference between max and idle 
power ranges from just 5% to 20%. In 
some cases, depending on the 
configuration of the storage product 
and the size of the power supply 
chosen the single power supply may 
have limited power overhead and it 
may not be desirable to routinely 
operate on a single power supply. 
The use of a single power supply is 
only desirable as an emergency 
back-up on a power supply failure, 
not as a routine mode of operation. 


Storage systems for OL-3 and OL-4 
typically are not “off-the-shelf” 
models but are application specific. 


Current power supply technologies 
are not capable of meeting titanium 
requirements for multi-volt power 
supplies and platinum is a stretch 
because of the range and type of 
voltage conversions that must occur 
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within a multi-volt power supply. 

Task 6 General Storage In general, Digital Europe is Remove storage products from the scope of the current 

Comment Products concerned that the consultant and study and move them to a separate study activity. 

the directorate do not have an 
adequate understanding of storage If the consultant wishes to maintain the storage base case, 
product capabilities, functionality, then the base case should be done based on a controller 
and power profiles. The selection of matched with one or two drawers of individual drive types: 
a base case which does not SSD, Large form factor and small form factor. That would 
represent actual product provide 3 base cases, but it accurately represents how 
configurations and is focused on those products would be sold and allow the consultant to 
storage products sold to small and assess the material implications of the different drive 
medium enterprises for use in office | types available for the system. As noted in the task 5 
and data closet environments is document, mixed drive systems do not appear until you 
acceptable if that is the part of the have four or five drawers of drives. 
market that the directorate wants to 
address with the Lot 9 requirements. 
However, the base case is not 
representative of storage products 
installed in enterprise data centers. 

Task 6 General Selection of Based on the discussions at the Set the base cases products released to the market in 

Comment base case stakeholder meeting, there is an 2014, not 2012. 
systems increased concern about the choice 
from 2012. of the server and storage base case 














models that were released to the 
market in 2012. Available data 
demonstrates that server models 
released in 2014 have improved 
performance per watt capabilities 
with comparable or improved idle 
power levels as compared to 
products introduced to market in 
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2012. Given that the consultant is 
evaluating potential improvement in 
energy consumption for a regulation 
that will go into effect in 2016 or 2017 
savings calculations should be made 
against the server and storage 
equipment that will represent the 
preponderance of data center 
equipment at that time. Tying 
potential savings to the 2012 
products will significantly overstate 
potential savings as we have noted, 
and for which we have provided 
specifics, in our comments 
elsewhere. Our concern was further 
heightened by the statement of one 
consultant that it is appropriate for 
the commission to take into account 
“learning” or capacity/performance 
cost reductions that will accrue from 
future technology advances and the 
cost benefits of volume production 
of components. If the commission 
considers it appropriate to consider 
the savings projected from 
“learning” then the base cases 
should reflect the current state of 
server and storage product 
capabilities, not the capabilities 
represented by n-1 or n-2 technology 
generations. 








Task 6 





General 








Discussion 
of Idle Limits 





At the Stakeholders meeting the 
consultant showed a slide (page 14 





When developing the Task 7 document, the consultant 
should not focus solely on idle power limits, for the 
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Comment at of the task 7 presentation) of the idle | reasons explained in the comment. In fact, Digital Europe 

Stakeholder | limits for one and two socket servers | has repeatedly stated its belief that idle power limits are 
Meeting on from the ENERGY STAR® not an appropriate metric for server or storage energy 
April 24, requirements and the additional efficiency requirements. If it is determined that energy 
2015 proposed silver and gold level idle efficiency metrics are required for server and storage 














limits proposed for the National 
Sanitation Foundation’s proposed 
Environmental Leadership Standards 
for Servers. Digital Europe is very 
concerned that the consultant is 
referencing idle limits from two 
product energy efficiency programs 
designed to identify leadership or 
“top runner” products, both of which 
are voluntary, and one of which is 
still in draft. These concerns exist at 
two levels: 


1. As we stated in the comments to 
the Final Draft Task 3 document, 
Section 1.1.2.4, idle power is not 
a good indicator of energy 
efficiency. A low idle power limit 
is biased to servers with low 
power, low core count, and low 
frequency processors which 
have a relatively low compute 
capacity and workload capability. 
Server efficiency is a function of 
the compute capacity, the ability 
to proportion power use to 
workload demand, and the ability 
to maximize the output, 
functionality and/or utilization of 





products, then the threshold requirements need to: 


1. Be based on a performance/watt metric, specifically 
SPEC SERT, to properly characterize the functional 
capabilities of the product. 


2. Use the ENERGY STAR product categories to 
differentiate the product types under server and 
storage products. These product categories recognize 
specific differences in product performance 
capabilities and power profiles which affect the ability 
to compare the capabilities and energy consumption of 
the products. 


3. Recognize the complexity of server and storage 
products and avoid creating inadvertent market 
barriers to server and storage products which offer 
customers the best workload delivered per unit of 
energy consumed. 
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the server or storage product 
over time. In some cases, servers 
with a higher idle power demand 
are able to do significantly more 
work per unit of energy 
consumed. In our comments to 
the draft task documents, we 
provided the consultant with data 
on how workload can be 
quadrupled while power use is 
increased by 10% to 20% when 
average server utilization is 
increased. Limiting the market to 
low power servers will have 
unintended consequences, as it 
will result in the need for 
significantly more servers to 
perform the workload on a range 
of complex and compute 
intensive workloads. 


Any effective energy efficiency 
requirements has to include a 
performance per watt metric as 
the basis for setting threshold 
requirements. It cannot be based 
on idle limits. Digital Europe has 
recommended to the consultant 
and the commission that they 
designate the SERT metric for 
this purpose. Digital Europe has 
provided SERT metric data on 
over 90 server machine types 
and 500 machine type 
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configurations to the consultant 
for evaluation. 


3. Any thresholds developed 
proposed or developed under Lot 
9 will be thresholds set for 
market entry. These will be 
significantly different than 
thresholds set for “top runner” or 
“energy efficient, leadership” 
products. It is inappropriate to 
reference idle limits from these 
programs in discussion of 
potential thresholds for the Lot 9 
server requirements. 








Task 6 





General 
Comment 








Material 
Analysis 





Digital Europe is concerned by the 
consultant’s statement at the 
stakeholder meeting that increasing 
the product reuse rate from 25% to 
50%, as recommended by comments, 
did not generate the increase in 
environmental benefit that the 
consultant expected. Based on this, 
the consultant was exploring other 
material impact models to get an 
answer that fit their expected 
outcome. 


We are concerned that changes to a 
defined process are being 
considered because the process did 
not deliver the “expected” outcome. 
If one considers that material and 
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reuse impacts are 10% of the total 
impacts and that reuse increases the 
life of products and components by a 
factor of roughly 2 (estimated by 
increasing the useful life from 5 
years to 10 years), then the change in 
environmental impacts of a few 
percent makes sense. 











